<HTML>
<CENTER><A HREF = "Section_intro.html">Previous Section</A> - <A HREF = "http://sparta.sandia.gov">SPARTA WWW Site</A> - <A HREF = "Manual.html">SPARTA Documentation</A> - <A HREF = "Section_commands.html#comm">SPARTA Commands</A> - <A HREF = "Section_commands.html">Next Section</A> 
</CENTER>






<HR>

<H3>2. Getting Started 
</H3>
<P>This section describes how to build and run SPARTA, for both new and
experienced users.
</P>
2.1 <A HREF = "#start_1">What's in the SPARTA distribution</A><BR>
2.2 <A HREF = "#start_2">Making SPARTA</A><BR>
2.3 <A HREF = "#start_3">Making SPARTA with optional packages</A><BR>
2.4 <A HREF = "#start_4">Building SPARTA as a library</A><BR>
2.5 <A HREF = "#start_5">Running SPARTA</A><BR>
2.6 <A HREF = "#start_6">Command-line options</A><BR>
2.7 <A HREF = "#start_7">Screen output</A> <BR>

<HR>

<H4><A NAME = "start_1"></A>2.1 What's in the SPARTA distribution 
</H4>
<P>When you download SPARTA you will need to unzip and untar the
downloaded file with the following commands:
</P>
<PRE>gunzip sparta*.tar.gz 
tar xvf sparta*.tar 
</PRE>
<P>This will create a SPARTA directory containing two files and several
sub-directories:
</P>
<DIV ALIGN=center><TABLE  BORDER=1 >
<TR><TD >README</TD><TD > text file</TD></TR>
<TR><TD >LICENSE</TD><TD > the GNU General Public License (GPL)</TD></TR>
<TR><TD >bench</TD><TD > benchmark problems</TD></TR>
<TR><TD >data</TD><TD > files with species, collision, and reaction parameters</TD></TR>
<TR><TD >doc</TD><TD > documentation</TD></TR>
<TR><TD >examples</TD><TD > simple test problems</TD></TR>
<TR><TD >python</TD><TD > Python wrapper</TD></TR>
<TR><TD >src</TD><TD > source files</TD></TR>
<TR><TD >tools</TD><TD > pre- and post-processing tools 
</TD></TR></TABLE></DIV>

<HR>

<H4><A NAME = "start_2"></A>2.2 Making SPARTA 
</H4>
<P>This section has the following sub-sections:
</P>
<UL><LI><A HREF = "#start_2_1">Read this first</A>
<LI><A HREF = "#start_2_2_1">Steps to build a SPARTA executable using make</A>
<LI><A HREF = "#start_2_2_2">Steps to build a SPARTA executable using CMake</A>
<LI><A HREF = "#start_2_3">Common errors that can occur when making SPARTA</A>
<LI><A HREF = "#start_2_4_1">Additional build tips using make</A>
<LI><A HREF = "#start_2_4_2">Additional build tips using CMake</A>
<LI><A HREF = "#start_2_5">Building for a Mac</A>
<LI><A HREF = "#start_2_6">Building for Windows</A> 
</UL>
<HR>

<A NAME = "start_2_1"></A><B><I>Read this first:</I></B> 

<P>Building SPARTA can be non-trivial.  You may need to edit a makefile,
there are compiler options to consider, additional libraries can be
used (MPI, JPEG).
</P>
<P>Please read this section carefully.  If you are not comfortable with
cmake, makefiles, or building codes on a Linux platform, or running an MPI
job on your machine, please find a local expert to help you.
</P>
<P>If you have a build problem that you are convinced is a SPARTA issue
(e.g. the compiler complains about a line of SPARTA source code), then
please send an email to the
<A HREF = "http://sparta.sandia.gov/authors.html">developers</A>.
</P>
<P>If you succeed in building SPARTA on a new kind of machine, for which
there isn't a similar Makefile in the src/MAKE directory or .cmake file
in cmake/presets, send it to the 
<A HREF = "http://sparta.sandia.gov/authors.html">developers</A> and we'll include it in future SPARTA releases.
</P>
<HR>

<A NAME = "start_2_2_1"></A><B><I>Steps to build a SPARTA executable using make:</I></B> 

<P><B>Step 0</B>
</P>
<P>The src directory contains the C++ source and header files for SPARTA.
It also contains a top-level Makefile and a MAKE sub-directory with
low-level Makefile.* files for many machines.  From within the src
directory, type "make" or "gmake".  You should see a list of available
choices.  If one of those is the machine and options you want, you can
type a command like:
</P>
<PRE>make g++
or
gmake mac 
</PRE>
<P>Note that on a multi-core platform you can launch a parallel make, by
using the "-j" switch with the make command, which will build SPARTA
more quickly.
</P>
<P>If you get no errors and an executable like spa_g++ or spa_mac is
produced, you're done; it's your lucky day.
</P>
<P>Note that by default none of the SPARTA optional packages are
installed.  To build SPARTA with optional packages, see <A HREF = "#start_3">this
section</A> below.
</P>
<P><B>Step 1</B>
</P>
<P>If Step 0 did not work, you will need to create a low-level Makefile
for your machine, like Makefile.foo.  Copy an existing
src/MAKE/Makefile.* as a starting point.  The only portions of the
file you need to edit are the first line, the "compiler/linker
settings" section, and the "SPARTA-specific settings" section.
</P>
<P><B>Step 2</B>
</P>
<P>Change the first line of src/MAKE/Makefile.foo to list the word "foo"
after the "#", and whatever other options it will set.  This is the
line you will see if you just type "make".
</P>
<P><B>Step 3</B>
</P>
<P>The "compiler/linker settings" section lists compiler and linker
settings for your C++ compiler, including optimization flags.  You can
use g++, the open-source GNU compiler, which is available on all Linux
systems.  You can also use mpicc which will typically be available if
MPI is installed on your system, though you should check which actual
compiler it wraps.  Vendor compilers often produce faster code.  On
boxes with Intel CPUs, we suggest using the commercial Intel icc
compiler, which can be downloaded from <A HREF = "http://www.intel.com/software/products/noncom">Intel's compiler site</A>.
</P>


<P>If building a C++ code on your machine requires additional libraries,
then you should list them as part of the LIB variable.
</P>
<P>The DEPFLAGS setting is what triggers the C++ compiler to create a
dependency list for a source file.  This speeds re-compilation when
source (*.cpp) or header (*.h) files are edited.  Some compilers do
not support dependency file creation, or may use a different switch
than -D.  GNU g++ works with -D.  Note that when you build SPARTA for
the first time on a new platform, a long list of *.d files will be
printed out rapidly.  This is not an error; it is the Makefile doing
its normal creation of dependencies.
</P>
<P><B>Step 4</B>
</P>
<P>The "system-specific settings" section has several parts.  Note that
if you change any -D setting in this section, you should do a full
re-compile, after typing "make clean", which will describe different
clean options.
</P>
<P>The SPA_INC variable is used to include options that turn on ifdefs
within the SPARTA code.  The options that are currently recognized are:
</P>
<UL><LI>-DSPARTA_GZIP
<LI>-DSPARTA_JPEG
<LI>-DSPARTA_PNG
<LI>-DSPARTA_FFMPEG
<LI>-DSPARTA_MAP
<LI>-DSPARTA_UNORDERED_MAP
<LI>-DSPARTA_SMALL
<LI>-DSPARTA_BIG
<LI>-DSPARTA_BIGBIG
<LI>-DSPARTA_LONGLONG_TO_LONG 
</UL>
<P>The read_data and dump commands will read/write gzipped files if you
compile with -DSPARTA_GZIP.  It requires that your Linux support the
"popen" command.
</P>
<P>If you use -DSPARTA_JPEG and/or -DSPARTA_PNG, the <A HREF = "dump.html">dump
image</A> command will be able to write out JPEG and/or PNG
image files respectively. If not, it will only be able to write out
PPM image files.  For JPEG files, you must also link SPARTA with a
JPEG library, as described below.  For PNG files, you must also link
SPARTA with a PNG library, as described below.
</P>
<P>If you use -DSPARTA_FFMPEG, the <A HREF = "dump_image.html">dump movie</A> command
will be available to support on-the-fly generation of rendered movies
the need to store intermediate image files.  It requires that your
machines supports the "popen" function in the standard runtime library
and that an FFmpeg executable can be found by SPARTA during the run.
</P>
<P>If you use -DSPARTA_MAP, SPARTA will use the STL map class for hash
tables.  This is less efficient than the unordered map class which is
not yet supported by all C++ compilers.  If you use
-DSPARTA_UNORDERED_MAP, SPARTA will use the unordered_map class for
hash tables and will assume it is part of the STL (e.g. this works for
Clang++).  The default is to use the unordered map class from the
"tri1" extension to the STL which is supported by most compilers.  So
only use either of these options if the build complains that unordered
maps are not recognized.
</P>
<P>Use at most one of the -DSPARTA_SMALL, -DSPARTA_BIG, -DSPARTA_BIGBIG
settings.  The default is -DSPARTA_BIG.  These refer to use of 4-byte
(small) vs 8-byte (big) integers within SPARTA, as described in
src/spatype.h.  The only reason to use the BIGBIG setting is if you
have a regular grid with more than ~2 billion grid cells or a
hierarchical grid with enough levels that grid cell IDs cannot fit in
a 32-bit integer.  In either case, SPARTA will generate an error
message for "Cell ID has too many bits".  See <A HREF = "Section_howto.html#howto_8">Section
4.8</A> of the manual for details on how cell
IDs are formatted.  The only reason to use the SMALL setting is if
your machine does not support 64-bit integers.
</P>
<P>In all cases, the size of problem that can be run on a per-processor
basis is limited by 4-byte integer storage to about 2 billion
particles per processor (2^31), which should not normally be a
restriction since such a problem would have a huge per-processor
memory and would run very slowly in terms of CPU secs/timestep.
</P>
<P>The -DSPARTA_LONGLONG_TO_LONG setting may be needed if your system or
MPI version does not recognize "long long" data types.  In this case a
"long" data type is likely already 64-bits, in which case this setting
will use that data type.
</P>
<P>Using one of the -DPACK_ARRAY, -DPACK_POINTER, and -DPACK_MEMCPY
options can make for faster parallel FFTs on some platforms.  The
-DPACK_ARRAY setting is the default.  See the <A HREF = "compute_fft_grid.html">compute
fft/grid</A> command for info about FFTs.  See Step
6 below for info about building SPPARKS with an FFT library.
</P>
<P><B>Step 5</B>
</P>
<P>The 3 MPI variables are used to specify an MPI library to build SPARTA
with. 
</P>
<P>If you want SPARTA to run in parallel, you must have an MPI library
installed on your platform.  If you use an MPI-wrapped compiler, such
as "mpicc" to build, you should be able to leave these 3 variables
blank; the MPI wrapper knows where to find the needed files.  If not,
and MPI is installed on your system in the usual place (under
/usr/local), you also may not need to specify these 3 variables.  On
some large parallel machines which use "modules" for their
compile/link environements, you may simply need to include the correct
module in your build environment.  Or the parallel machine may have a
vendor-provided MPI which the compiler has no trouble finding.
</P>
<P>Failing this, with these 3 variables you can specify where the mpi.h
file is found (via MPI_INC), and the MPI library file is found (via
MPI_PATH), and the name of the library file (via MPI_LIB).  See
Makefile.serial for an example of how this can be done.
</P>
<P>If you are installing MPI yourself, we recommend MPICH 1.2 or 2.0 or
OpenMPI.  MPICH can be downloaded from the <A HREF = "http://www-unix.mcs.anl.gov/mpi">Argonne MPI
site</A>.  OpenMPI can be downloaded the
<A HREF = "http://www.open-mpi.org">OpenMPI site</A>.  If you are running on a big
parallel platform, your system admins or the vendor should have
already installed a version of MPI, which will be faster than MPICH or
OpenMPI, so find out how to build and link with it.  If you use MPICH
or OpenMPI, you will have to configure and build it for your platform.
The MPI configure script should have compiler options to enable you to
use the same compiler you use for the SPARTA build, which can avoid
problems that can arise when linking SPARTA to the MPI library.
</P>
<P>If you just want to run SPARTA on a single processor, you can use the
dummy MPI library provided in src/STUBS, since you don't need a true
MPI library installed on your system.  You will also need to build the
STUBS library for your platform before making SPARTA itself.  From the
src directory, type "make mpi-stubs", or from within the STUBS dir,
type "make" and it should create a libmpi.a suitable for linking to
SPARTA.  If this build fails, you will need to edit the STUBS/Makefile
for your platform.
</P>
<P>The file STUBS/mpi.cpp provides a CPU timer function called
MPI_Wtime() that calls gettimeofday() .  If your system doesn't
support gettimeofday() , you'll need to insert code to call another
timer.  Note that the ANSI-standard function clock() function rolls
over after an hour or so, and is therefore insufficient for timing
long SPARTA simulations.
</P>
<P><B>Step 6</B>
</P>
<P>The 3 FFT variables allow you to specify an FFT library which SPARTA
uses (for performing 1d FFTs) when built with its FFT package, which
contains commands that invoke FFTs.
</P>
<P>SPARTA supports various open-source or vendor-supplied FFT libraries
for this purpose.  If you leave these 3 variables blank, SPARTA will
use the open-source <A HREF = "http://kissfft.sf.net">KISS FFT library</A>, which is
included in the SPARTA distribution.  This library is portable to all
platforms and for typical SPARTA simulations is almost as fast as FFTW
or vendor optimized libraries.  If you are not including the FFT
package in your build, you can also leave the 3 variables blank.
</P>
<P>Otherwise, select which kinds of FFTs to use as part of the FFT_INC
setting by a switch of the form -DFFT_XXX. 
Recommended values for XXX
are: MKL or FFTW3.  FFTW2 and NONE are supported as legacy options.
Selecting -DFFT_FFTW will use the FFTW3 library and -DFFT_NONE will
use the KISS library described above.
described above.
</P>
<P>You may also need to set the FFT_INC, FFT_PATH, and FFT_LIB variables,
so the compiler and linker can find the needed FFT header and library
files.  Note that on some large parallel machines which use "modules"
for their compile/link environements, you may simply need to include
the correct module in your build environment.  Or the parallel machine
may have a vendor-provided FFT library which the compiler has no
trouble finding.
</P>
<P>FFTW is a fast, portable library that should also work on any
platform.  You can download it from
<A HREF = "http://www.fftw.org">www.fftw.org</A>.  Both the legacy version 2.1.X and
the newer 3.X versions are supported as -DFFT_FFTW2 or -DFFT_FFTW3.
Building FFTW for your box should be as simple as ./configure; make.
Note that on some platforms FFTW2 has been pre-installed, and uses
renamed files indicating the precision it was compiled with,
e.g. sfftw.h, or dfftw.h instead of fftw.h.  In this case, you can
specify an additional define variable for FFT_INC called -DFFTW_SIZE,
which will select the correct include file.  In this case, for FFT_LIB
you must also manually specify the correct library, namely -lsfftw or
-ldfftw.
</P>
<P>The FFT_INC variable also allows for a -DFFT_SINGLE setting that will
use single-precision FFTs, which can speed-up the calculation,
particularly in parallel or on GPUs.  Fourier transform and related
PPPM operations are somewhat insensitive to floating point truncation
errors and thus do not always need to be performed in double
precision.  Using the -DFFT_SINGLE setting trades off a little
accuracy for reduced memory use and parallel communication costs for
transposing 3d FFT data.
</P>
<P><B>Step 7</B>
</P>
<P>The 3 JPG variables allow you to specify a JPEG and/or PNG library
which SPARTA uses when writing out JPEG or PNG files via the <A HREF = "dump_image.html">dump
image</A> command. These can be left blank if you do not
use the -DSPARTA_JPEG or -DSPARTA_PNG switches discussed above in Step
4, since in that case JPEG/PNG output will be disabled.
</P>
<P>A standard JPEG library usually goes by the name libjpeg.a or
libjpeg.so and has an associated header file jpeglib.h. Whichever JPEG
library you have on your platform, you'll need to set the appropriate
JPG_INC, JPG_PATH, and JPG_LIB variables, so that the compiler and
linker can find it.
</P>
<P>A standard PNG library usually goes by the name libpng.a or libpng.so
and has an associated header file png.h. Whichever PNG library you
have on your platform, you'll need to set the appropriate JPG_INC,
JPG_PATH, and JPG_LIB variables, so that the compiler and linker can
find it.
</P>
<P>As before, if these header and library files are in the usual place on
your machine, you may not need to set these variables.
</P>
<P><B>Step 8</B>
</P>
<P>Note that by default none of the SPARTA optional packages are
installed.  To build SPARTA with optional packages, see <A HREF = "#start_3">this
section</A> below, before proceeding to Step 9.
</P>
<P><B>Step 9</B>
</P>
<P>That's it.  Once you have a correct Makefile.foo, and you have
pre-built any other needed libraries (e.g. MPI), all you need to do
from the src directory is type one of the following:
</P>
<PRE>make foo
make -j N foo
gmake foo
gmake -j N foo 
</PRE>
<P>The -j or -j N switches perform a parallel build which can be much
faster, depending on how many cores your compilation machine has.  N
is the number of cores the build runs on.
</P>
<P>You should get the executable spa_foo when the build is complete.
</P>
<HR>

<A NAME = "start_2_2_2"></A><B><I>Steps to build a SPARTA executable using CMake:</I></B> 

<P><B>Step 0</B>
</P>
<P>Please review https://github.com/sparta/sparta/blob/master/BUILD_CMAKE.md and ensure that
CMake version 3.10.0 or greater is installed:
</P>
<PRE>which cmake
which cmake3
cmake --version 
</PRE>
<P>On clusters and supercomputers one can use modules to load cmake:
</P>
<PRE>module avail cmake
module load <CMAKE> 
</PRE>
<P>On Linux one may use apt, yum, or pacman to install cmake.
</P>
<P>On Mac one may use brew or macports to install cmake.
</P>
<P><B>Step 1</B>
</P>
<P>The cmake directory contains the CMake source files for SPARTA. Create a build
directory and from within the build directory, run cmake:
</P>
<PRE>mkdir build
cd build
cmake -LH -DSPARTA_MACHINE=tutorial /path/to/sparta/cmake 
</PRE>
<P>This will generate the default Makefiles and print the SPARTA CMake options. To
list the generated targets, do:
</P>
<PRE>make help 
</PRE>
<P>Now you can try to build the SPARTA binaries with:
</P>
<PRE>make 
</PRE>
<P>If everything works, an executable named spa_tutorial and a library named
libsparta.a will be produced in build/src.
</P>
<P><B>Step 2</B>
</P>
<P>If Step 1 did not work, see if you can use any system presets from
/path/to/sparta/cmake/presets. To select a preset:
</P>
<P>cd build
</P>
<P># Clear the CMake files
rm -rf CMake*
</P>
<PRE>cmake -C /path/to/sparta/cmake/presets/<NAME>.cmake -DSPARTA_MACHINE=tutorial /path/to/sparta/cmake
make 
</PRE>
<P><B>Step 3</B>
</P>
<P>If Step 2 did not work, look at cmake -LH for a list of SPARTA CMake options and their
meaning, then modify one or more of those options by doing:
</P>
<PRE>cd build
rm -rf CMake*
cmake -C /path/to/sparta/cmake/presets/<NAME>.cmake -D<OPTION_NAME>=<VALUE> /path/to/sparta/cmake
make 
</PRE>
<P>where <OPTION_NAME> and <VALUE> correspond to valid option value pairs listed by
cmake -LH. For the SPARTA_DEFAULT_CXX_COMPILE_FLAGS option, see Step 4.
</P>
<P>For a full list of CMake option value pairs, see cmake -LAH. The most relevant
CMake options (with example values) for our purposes here are:
</P>
<P>-DCMAKE_C_COMPILER=gcc
-DCMAKE_CXX_COMPILER=/usr/local/bin/g++
-DCMAKE_CXX_FLAGS=-O3
</P>
<P>If your cmake command line is getting too long, consider placing it in a bash
script and escaping newlines. For example:
</P>
<PRE>cmake -C  /path/to/sparta/cmake/presets/<NAME>.cmake -D  -D<OPTION_NAME>=<VALUE> /path/to/sparta/cmake 
</PRE>
<P><B>Step 4</B>
</P>
<P>The SPARTA_DEFAULT_CXX_COMPILE_FLAGS option passes flags to the compiler when
building object files.  Note that if you change any -D setting in this section,
you should do a full re-compile, after typing "make clean".
</P>
<P>The SPARTA_DEFAULT_CXX_COMPILE_FLAGS option is typically used to include options
that turn on ifdefs within the SPARTA code.  The options that are currently recogized are:
</P>
<UL><LI>-DSPARTA_GZIP
<LI>-DSPARTA_JPEG
<LI>-DSPARTA_PNG
<LI>-DSPARTA_FFMPEG
<LI>-DSPARTA_MAP
<LI>-DSPARTA_UNORDERED_MAP
<LI>-DSPARTA_SMALL
<LI>-DSPARTA_BIG
<LI>-DSPARTA_BIGBIG
<LI>-DSPARTA_LONGLONG_TO_LONG 
</UL>
<P>The read_data and dump commands will read/write gzipped files if you
compile with -DSPARTA_GZIP.  It requires that your Linux support the
"popen" command.
</P>
<P>If you use -DSPARTA_JPEG and/or -DSPARTA_PNG, the <A HREF = "dump.html">dump
image</A> command will be able to write out JPEG and/or PNG
image files respectively. If not, it will only be able to write out
PPM image files.  For JPEG files, you must also link SPARTA with a
JPEG library, as described below.  For PNG files, you must also link
SPARTA with a PNG library, as described below.
</P>
<P>If you use -DSPARTA_FFMPEG, the <A HREF = "dump_image.html">dump movie</A> command
will be available to support on-the-fly generation of rendered movies
the need to store intermediate image files.  It requires that your
machines supports the "popen" function in the standard runtime library
and that an FFmpeg executable can be found by SPARTA during the run.
</P>
<P>If you use -DSPARTA_MAP, SPARTA will use the STL map class for hash
tables.  This is less efficient than the unordered map class which is
not yet supported by all C++ compilers.  If you use
-DSPARTA_UNORDERED_MAP, SPARTA will use the unordered_map class for
hash tables and will assume it is part of the STL (e.g. this works for
Clang++).  The default is to use the unordered map class from the
"tri1" extension to the STL which is supported by most compilers.  So
only use either of these options if the build complains that unordered
maps are not recognized.
</P>
<P>Use at most one of the -DSPARTA_SMALL, -DSPARTA_BIG, -DSPARTA_BIGBIG
settings.  The default is -DSPARTA_BIG.  These refer to use of 4-byte
(small) vs 8-byte (big) integers within SPARTA, as described in
src/spatype.h.  The only reason to use the BIGBIG setting is if you
have a regular grid with more than ~2 billion grid cells or a
hierarchical grid with enough levels that grid cell IDs cannot fit in
a 32-bit integer.  In either case, SPARTA will generate an error
message for "Cell ID has too many bits".  See <A HREF = "Section_howto.html#howto_8">Section
4.8</A> of the manual for details on how cell
IDs are formatted.  The only reason to use the SMALL setting is if
your machine does not support 64-bit integers.
</P>
<P>In all cases, the size of problem that can be run on a per-processor
basis is limited by 4-byte integer storage to about 2 billion
particles per processor (2^31), which should not normally be a
restriction since such a problem would have a huge per-processor
memory and would run very slowly in terms of CPU secs/timestep.
</P>
<P>The -DSPARTA_LONGLONG_TO_LONG setting may be needed if your system or
MPI version does not recognize "long long" data types.  In this case a
"long" data type is likely already 64-bits, in which case this setting
will use that data type.
</P>
<P>Using one of the -DPACK_ARRAY, -DPACK_POINTER, and -DPACK_MEMCPY
options can make for faster parallel FFTs on some platforms.  The
-DPACK_ARRAY setting is the default.  See the <A HREF = "compute_fft_grid.html">compute
fft/grid</A> command for info about FFTs.  See STEP
??? below for info about building SPPARKS with an FFT library.
</P>
<P><B>Step 5</B>
</P>
<P>This step is optional. Once you get Steps 3 and 4 working by modifying the
options to the cmake command, try setting the same options in
/path/to/sparta/cmake/presets/<NEW>.cmake by copying 
/path/to/sparta/cmake/presets/<NAME>.cmake and modifying the cmake
source code. Note that the CMake cache is sticky and will only evict a 
cached option value pair if you use -D or the FORCE argument to CMake's set
routine.
</P>
<P>Now just do:
</P>
<PRE>cd build
rm -rf CMake*
cmake -C /path/to/sparta/cmake/presets/<NEW>.cmake /path/to/sparta/cmake
make 
</PRE>
<P>consider sharing and vetting <NEW>.cmake by opening a pull request at
https://github.com/sparta/sparta/.
</P>
<P><B>Step 6</B>
</P>
<P>This step explains how to enable and select MPI in the SPARTA CMake
configuration. There may already be a preset in 
/path/to/sparta/cmake/presets that selects the correct MPI installation.
</P>
<P>By default, SPARTA configures with MPI enabled and cmake will print which MPI
was selected. To build serial binaries, use SPARTA's MPI_STUBS package:
</P>
<PRE>cmake -DPKG_MPI_STUBS=ON /path/to/sparta/cmake 
</PRE>
<P>You may want a different MPI installation than CMake finds. CMake uses module
files such as FindMPI.cmake to handle wiring in a given installation of a 
library and its headers. If you're on a cluster or supercomputer, use module 
before running cmake so that cmake finds the MPI installation you'd like to
use:
</P>
<P># Show which modules are loaded
module list
</P>
<P># Show which modules are available
module avail
</P>
<PRE>module load <MPI> 
</PRE>
<P>On Linux one may use apt, yum, or pacman to install MPI.
</P>
<P>On Mac one may use brew or macports to install MPI.
</P>
<P>Verify that cmake found the correct MPI installation:
</P>
<P>cd build
rm -rf CMake*
</P>
<PRE># cmake should print "Found MPI*" strings
cmake <B>options</B> /path/to/sparta/cmake 
</PRE>
<P>Note that if the preset file you're using enables PKG_MPI_STUBS, MPI will not be
searched for unless you explicitly disable PKG_MPI_STUBS in the preset file.
</P>
<P>If you'd like to use a custom MPI installation or cmake is not locating the MPI
installation you've selected via the module command or package manager, try
export MPI_ROOT=/path/to/mpi/install before running cmake. Otherwise, please see
https://cmake.org/cmake/help/v3.10/module/FindMPI.html#variables-for-locating-mpi.
Note that this documentation link is for CMake version 3.10.
</P>
<P><B>Step 7</B>
</P>
<P>You may select between 3 thiry party libraries (TPL) for FFT which SPARTA uses when
configured with cmake -DFFT=<I>FFTW2,FFTW3,MKL</I>. SPARTA also provides a FFT
package which can be selected with cmake -DPKG_FFT=ON.
</P>
<P>You may need to install the FFT TPL you're interested in using. If you're on a
cluster or supercomputer, use module before running cmake so that cmake finds
the FFT installation you'd like to use:
</P>
<P># Show which modules are loaded
module list
</P>
<P># Show which modules are available
module avail
</P>
<PRE>module load <FFT> 
</PRE>
<P>On Linux one may use apt, yum, or pacman to install FFT.
</P>
<P>On Mac one may use brew or macports to install FFT.
</P>
<P>Verify that cmake found the correct MPI installation:
</P>
<P>cd build
rm -rf CMake*
</P>
<PRE># cmake should print "Found FFT*" strings
cmake <B>options</B> /path/to/sparta/cmake 
</PRE>
<P>Note that if the preset file you're using enables PKG_FFT, FFT will not be
searched for unless you explicitly disable PKG_FFT in the preset file.
</P>
<P>If you'd like to use a custom FFT installation or cmake is not locating the FFT
installation you've selected via the module command or package manager, try
export FFT_ROOT=/path/to/fft/install before running cmake. Otherwise, please
open an issue at https://github.com/sparta/sparta/issues.
</P>
<P><B>Step 8</B>
</P>
<P>You may select between 2 TPLs, JPEG or PNG, for writing out JPEG or PNG files
via the <A HREF = "dump_image.html">dump image</A> command. To select a TPL, use:
</P>
<PRE>cmake -DBUILD_JPEG=ON /path/to/sparta/cmake 
</PRE>
<P>or:
</P>
<PRE>cmake -DBUILD_PNG=ON /path/to/sparta/cmake 
</PRE>
<P>If you'd like to use a custom jpeg or png installation, please see 
https://cmake.org/cmake/help/v3.10/module/FindJPEG.html or
https://cmake.org/cmake/help/v3.10/module/FindPNG.html. Note that these
documentation links are for CMake version 3.10.
</P>
<P><B>Step 9</B>
</P>
<P>By default, none of the SPARTA optional packages are installed. To build SPARTA
with optional packages, use:
</P>
<PRE>cmake -DPKG_XXX=ON /path/to/sparta/cmake 
</PRE>
<P>Where XXX is the package to enable. For a full list of optional packages, see:
</P>
<PRE>cmake -LH /path/to/sparta/cmake 
</PRE>
<P><B>Step 10</B>
</P>
<P>Once you have a correct cmake command line or the <NAME>.cmake preset file, just
do:
</P>
<PRE>cd build
cmake <B>OPTIONS</B> /path/to/sparta/cmake 
</PRE>
<P>or:
</P>
<P>cd build
cmake -C /path/to/sparta/cmake/presets/<NAME>.cmake -DSPARTA_MACHINE=tutorial /path/to/sparta/cmake
</P>
<PRE>make -j N 
</PRE>
<P>The -j or -j N switches perform a parallel build which can be much faster, 
depending on how many cores your compilation machine has. N is the number of
cores the build runs on.
</P>
<P>You should get build/src/spa_tutorial and build/src/libsparta.a.
</P>
<HR>

<A NAME = "start_2_3"></A><B><I>Errors that can occur when making SPARTA:</I></B> 

<P>IMPORTANT NOTE: If an error occurs when building SPARTA, the compiler
or linker will state very explicitly what the problem is.  The error
message should give you a hint as to which of the steps above has
failed, and what you need to do in order to fix it.  Building a code
with a Makefile is a very logical process.  The compiler and linker
need to find the appropriate files and those files need to be
compatible with SPARTA source files.  When a make fails, there is
usually a very simple reason, which you or a local expert will need to
fix.
</P>
<P>Here are two non-obvious errors that can occur:
</P>
<P>(1) If the make command breaks immediately with errors that indicate
it can't find files with a "*" in their names, this can be because
your machine's native make doesn't support wildcard expansion in a
makefile.  Try gmake instead of make.  If that doesn't work, try using
a -f switch with your make command to use a pre-generated
Makefile.list which explicitly lists all the needed files, e.g.
</P>
<PRE>make makelist
make -f Makefile.list g++
gmake -f Makefile.list mac 
</PRE>
<P>The first "make" command will create a current Makefile.list with all
the file names in your src dir.  The 2nd "make" command (make or
gmake) will use it to build SPARTA.
</P>
<P>(2) If you get an error that says something like 'identifier "atoll"
is undefined', then your machine does not support "long long"
integers.  Try using the -DSPARTA_LONGLONG_TO_LONG setting described
above in Step 4.
</P>
<HR>

<A NAME = "start_2_4_1"></A><B><I>Additional build tips using make:</I></B> 

<P>(1) Building SPARTA for multiple platforms.
</P>
<P>You can make SPARTA for multiple platforms from the same src
directory.  Each target creates its own object sub-directory called
Obj_name where it stores the system-specific *.o files.
</P>
<P>(2) Cleaning up.
</P>
<P>Typing "make clean-all" or "make clean-foo" will delete *.o object
files created when SPARTA is built, for either all builds or for a
particular machine.
</P>
<HR>

<A NAME = "start_2_4_2"></A><B><I>Additional build tips using CMake:</I></B> 

<P>(1) Building SPARTA for multiple platforms.
</P>
<P>It's best to build SPARTA for multiple platforms from different
build directories. However, each target creates its own spa_TARGET binary and
multiple targets can be built from the same build directory. Note that the *.o
object files in build/src will reflective of the most recent build
configuration. Also note that if BUILD_SHARED_LIBS was enabled,
libsparta will be reflective of the most recent build configuration.
</P>
<P>(2) Cleaning up.
</P>
<P>Typing "make clean" will delete all binary files for the most recent build
configuration.
</P>
<HR>

<A NAME = "start_2_5"></A><B><I>Building for a Mac:</I></B> 

<P>OS X is BSD Unix, so it should just work.  See the Makefile.mac or
cmake/presets/mac.cmake file.
</P>
<HR>

<A NAME = "start_2_6"></A><B><I>Building for Windows:</I></B> 

<P>At some point we may provide a pre-built Windows executable
for SPARTA.  Until then you will need to build an executable from 
source files.
</P>
<P>One way to do this is install and use cygwin to build SPARTA with a
standard Linux make or CMake, just as you would on any Linux box.
</P>
<P>You can also import the *.cpp and *.h files into Microsoft Visual
Studio.  If someone does this and wants to provide project files or
other Windows build tips, please send them to the
<A HREF = "http://sparta.sandia.gov/authors.html">developers</A> and we will include
them in the distribution.
</P>
<HR>

<H4><A NAME = "start_3"></A>2.3 Making SPARTA with optional packages 
</H4>
<P>This section has the following sub-sections:
</P>
<P><A HREF = "#start_3_1">Package basics</A>
<A HREF = "#start_3_2_1">Including/excluding packages with make</A>
<A HREF = "#start_3_2_2">Including/excluding packages with CMake</A>
</P>
<HR>

<A NAME = "start_3_1"></A><B><I>Package basics:</I></B> 

<P>The source code for SPARTA is structured as a set of core files which
are always included, plus optional packages.  Packages are groups of
files that enable a specific set of features.  For example, the FFT
package which includes a <A HREF = "compute_fft_grid.html">compute fft/grid</A>
command and a 2d and 3d FFT library.
</P>
<P>For make:
You can see the list of all packages by typing "make package" from
within the src directory of the SPARTA distribution. This also lists
various make commands that can be used to manipulate packages.
</P>
<P>For CMake:
You can see the list of all packages by typing "cmake -DSPARTA_LIST_PKGS=ON"
from within the build directory.
</P>
<P>If you use a command in a SPARTA input script that is part of a
package, you must have built SPARTA with that package, else you will
get an error that the style is invalid or the command is unknown.
Every command's doc page specfies if it is part of a package.
</P>
<HR>

<A NAME = "start_3_2_1"></A><B><I>Including/excluding packages with make:</I></B> 

<P>To use (or not use) a package you must include it (or exclude it)
before building SPARTA.  From the src directory, this is typically as
simple as:
</P>
<PRE>make yes-fft
make g++ 
</PRE>
<P>or
</P>
<PRE>make no-fft
make g++ 
</PRE>
<P>NOTE: You should NOT include/exclude packages and build SPARTA in a
single make command using multiple targets, e.g. make yes-fft g++.
This is because the make procedure creates a list of source files that
will be out-of-date for the build if the package configuration changes
within the same command.
</P>
<P>Some packages have individual files that depend on other packages
being included.  SPARTA checks for this and does the right thing.
I.e. individual files are only included if their dependencies are
already included.  Likewise, if a package is excluded, other files
dependent on that package are also excluded.
</P>
<P>If you will never run simulations that use the features in a
particular packages, there is no reason to include it in your build.
</P>
<P>When you download a SPARTA tarball, no packages are pre-installed in
the src directory.
</P>
<P>Packages are included or excluded by typing "make yes-name" or "make
no-name", where "name" is the name of the package in lower-case, e.g.
name = fft for the FFT package.  You can also type "make yes-all", or
"make no-all" to include/exclude all packages.  Type "make package" to
see all of the package-related make options.
</P>
<P>NOTE: Inclusion/exclusion of a package works by simply moving files
back and forth between the main src directory and sub-directories with
the package name (e.g. src/FFT or src/KOKKOS), so that the files are
seen or not seen when SPARTA is built.  After you have included or
excluded a package, you must re-build SPARTA.
</P>
<P>Additional package-related make options exist to help manage SPARTA
files that exist in both the src directory and in package
sub-directories.  You do not normally need to use these commands
unless you are editing SPARTA files.
</P>
<P>Typing "make package-update" or "make pu" will overwrite src files
with files from the package sub-directories if the package has been
included.  It should be used after a patch is installed, since patches
only update the files in the package sub-directory, but not the src
files.  Typing "make package-overwrite" will overwrite files in the
package sub-directories with src files.
</P>
<P>Typing "make package-status" or "make ps" will show which packages are
currently included. For those that are included, it will list any
files that are different in the src directory and package
sub-directory.  Typing "make package-diff" lists all differences
between these files.  Again, type "make package" to see all of the
package-related make options.
</P>
<HR>

<A NAME = "start_3_2_2"></A><B><I>Including/excluding packages with CMake:</I></B> 

<P>To use (or not use) a package you must include it (or exclude it)
before building SPARTA.  From the build directory, do:
</P>
<PRE>cmake -DPKG_FFT=ON /path/to/sparta/cmake
make -j 
</PRE>
<P>or
</P>
<PRE>cmake -DPKG_FFT=OFF /path/to/sparta/cmake
make -j 
</PRE>
<P>Some packages have individual files that depend on other packages
being included.  SPARTA checks for this and does the right thing.
I.e. individual files are only included if their dependencies are
already included.  Likewise, if a package is excluded, other files
dependent on that package are also excluded.
</P>
<P>If you will never run simulations that use the features in a
particular packages, there is no reason to include it in your build.
</P>
<P>When you download a SPARTA tarball, no packages are pre-installed in
the build/src directory.
</P>
<P>Packages are included or excluded by typing "cmake -DPKG_NAME=ON" or 
"cmake -DPKG_NAME=OFF", where "NAME" is the name of the package in upper-case, 
e.g. name = FFT for the FFT package. You can also type "cmake
-DSPARTA_ENABLE_ALL_PKGS=ON", or "cmake -DSPARTA_DISABLE_ALL_PKGS=ON" to 
include or exclude all packages. Type "cmake -DSPARTA_LIST_PKGS=ON" to
see all of the package-related CMake options.
</P>
<P>NOTE: Inclusion or exclusion of a package works by setting CMake boolean
variables to generate the correct Makefile targets and dependencies. After you
have included or excluded a package, you must re-build SPARTA.
</P>
<P>If a SPARTA package has source code changes, simply run "make" to rebuild SPARTA
with these changes.
</P>
<P>Typing "cmake" from the build directory will show which packages are currently
included.
</P>
<HR>

<H4><A NAME = "start_4"></A>2.4 Building SPARTA as a library 
</H4>
<P>SPARTA can be built as either a static or shared library, which can
then be called from another application or a scripting language.  See
<A HREF = "Section_howto.html#howto_7">Section 6.7</A> for more info on coupling
SPARTA to other codes.  See <A HREF = "Section_python.html">Section 11</A> for more
info on wrapping and running SPARTA from Python.
</P>
<P>The CMake build system will produce the library static of dynamic libsparta
library in build/src.
</P>
<H5><B>Static library:</B> 
</H5>
<P>CMake builds sparta as a static library in libsparta.a, by default.
</P>
<P>To build SPARTA as a static library (*.a file on Linux), type
</P>
<PRE>make foo mode=lib 
</PRE>
<P>where foo is the machine name.  This kind of library is typically used
to statically link a driver application to SPARTA, so that you can
insure all dependencies are satisfied at compile time.  This will use
the ARCHIVE and ARFLAGS settings in src/MAKE/Makefile.foo.  The build
will create the file libsparta_foo.a which another application can
link to.  It will also create a soft link libsparta.a, which will
point to the most recently built static library.
</P>
<H5><B>Shared library:</B> 
</H5>
<P>To build SPARTA as a shared library (*.so file on Linux), which can be
dynamically loaded, e.g. from Python, type
</P>
<PRE>make foo mode=shlib 
</PRE>
<P>or:
</P>
<PRE>cmake -C /path/to/sparta/cmake/presets/foo.cmake -DBUILD_SHARED_LIBS=ON /path/to/sparta/cmake
make 
</PRE>
<P>where foo is the machine name.  This kind of library is required when
wrapping SPARTA with Python; see <A HREF = "Section_python.html">Section_python</A>
for details.  This will use the SHFLAGS and SHLIBFLAGS settings in
src/MAKE/Makefile.foo and perform the build in the directory
Obj_shared_foo.  This is so that each file can be compiled with the
-fPIC flag which is required for inclusion in a shared library.  The
build will create the file libsparta_foo.so which another application
can link to dyamically.  It will also create a soft link libsparta.so,
which will point to the most recently built shared library.  This is
the file the Python wrapper loads by default.
</P>
<P>Note that for a shared library to be usable by a calling program, all
the auxiliary libraries it depends on must also exist as shared
libraries.  This will be the case for libraries included with SPARTA,
such as the dummy MPI library in src/STUBS or any package libraries in
lib/packages, since they are always built as shared libraries using
the -fPIC switch.  However, if a library like MPI or FFTW does not
exist as a shared library, the shared library build will generate an
error.  This means you will need to install a shared library version
of the auxiliary library.  The build instructions for the library
should tell you how to do this.
</P>
<P>Here is an example of such errors when the system FFTW or provided
lib/colvars library have not been built as shared libraries:
</P>
<PRE>/usr/bin/ld: /usr/local/lib/libfftw3.a(mapflags.o): relocation
R_X86_64_32 against `.rodata' can not be used when making a shared
object; recompile with -fPIC
/usr/local/lib/libfftw3.a: could not read symbols: Bad value 
</PRE>
<PRE>/usr/bin/ld: ../../lib/colvars/libcolvars.a(colvarmodule.o):
relocation R_X86_64_32 against `__pthread_key_create' can not be used
when making a shared object; recompile with -fPIC
../../lib/colvars/libcolvars.a: error adding symbols: Bad value 
</PRE>
<P>As an example, here is how to build and install the <A HREF = "http://www-unix.mcs.anl.gov/mpi">MPICH
library</A>, a popular open-source version of MPI, distributed by
Argonne National Labs, as a shared library in the default
/usr/local/lib location:
</P>


<PRE>./configure --enable-shared
make
make install 
</PRE>
<P>You may need to use "sudo make install" in place of the last line if
you do not have write privileges for /usr/local/lib.  The end result
should be the file /usr/local/lib/libmpich.so.
</P>
<H5><B>Additional requirement for using a shared library:</B> 
</H5>
<P>The operating system finds shared libraries to load at run-time using
the environment variable LD_LIBRARY_PATH.
</P>
<P>Using CMake, ensure that CMAKE_INSTALL_PREFIX is set properly and then run "make
-j install" or add build/src to LD_LIBRARY_PATH in your shell's environment.
</P>
<P>Using make, you may wish to copy the file src/libsparta.so or 
src/libsparta_g++.so (for example) to a place the system can find it 
by default, such as /usr/local/lib, or you may wish to add the SPARTA
src directory to LD_LIBRARY_PATH, so that the current version of the 
shared library is always available to programs that use it.
</P>
<P>For the csh or tcsh shells, you would add something like this to your
~/.cshrc file:
</P>
<PRE>setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/home/sjplimp/sparta/src 
</PRE>
<H5><B>Calling the SPARTA library:</B> 
</H5>
<P>Either flavor of library (static or shared) allows one or more SPARTA
objects to be instantiated from the calling program.
</P>
<P>When used from a C++ program, all of SPARTA is wrapped in a SPARTA_NS
namespace; you can safely use any of its classes and methods from
within the calling code, as needed.
</P>
<P>When used from a C or Fortran program or a scripting language like
Python, the library has a simple function-style interface, provided in
src/library.cpp and src/library.h.
</P>
<P>See <A HREF = "Section_howto.html#howto_7">Section_howto 4.7</A> of the manual for
ideas on how to couple SPARTA to other codes via its library
interface.  See <A HREF = "Section_python.html">Section_python</A> of the manual for
a description of the Python wrapper provided with SPARTA that operates
through the SPARTA library interface.
</P>
<P>The files src/library.cpp and library.h define the C-style API for
using SPARTA as a library.  See <A HREF = "Section_howto.html#howto_6">Section_howto
4.6</A> of the manual for a description of the
interface and how to extend it for your needs.
</P>
<HR>

<H4><A NAME = "start_5"></A>2.5 Running SPARTA 
</H4>
<P>By default, SPARTA runs by reading commands from standard input.  Thus
if you run the SPARTA executable by itself, e.g.
</P>
<PRE>spa_g++ 
</PRE>
<P>it will simply wait, expecting commands from the keyboard.  Typically
you should put commands in an input script and use I/O redirection,
e.g.
</P>
<PRE>spa_g++ < in.file 
</PRE>
<P>For parallel environments this should also work.  If it does not, use
the '-in' command-line switch, e.g.
</P>
<PRE>spa_g++ -in in.file 
</PRE>
<P><A HREF = "Section_commands.html">Section 3</A> describes how input scripts are
structured and what commands they contain.
</P>
<P>You can test SPARTA on any of the sample inputs provided in the
examples or bench directory.  Input scripts are named in.* and sample
outputs are named log.*.name.P where name is a machine and P is the
number of processors it was run on.
</P>
<P>Here is how you might run one of the benchmarks on a
Linux box, using mpirun to launch a parallel job:
</P>
<P>cd src
make g++
cp spa_g++ ../bench
cd ../bench
mpirun -np 4 spa_g++ < in.free
</P>
<P>or:
</P>
<PRE>cd build
cmake -DCMAKE_CXX_COMPILER=g++ -DSPARTA_MACHINE=g++ /path/to/sparta/cmake
cp src/spa_g++ /path/to/bench
cd /path/to/bench
mpirun -np 4 spa_g++ < in.free 
</PRE>
<P>See <A HREF = "http://sparta.sandia.gov/bench.html">this page</A> for timings for this and the other benchmarks on
various platforms.
</P>


<P>The screen output from SPARTA is described in the next section.  As it
runs, SPARTA also writes a log.sparta file with the same information.
</P>
<P>Note that this sequence of commands copies the SPARTA executable
(spa_g++) to the directory with the input files.  This may not be
necessary, but some versions of MPI reset the working directory to
where the executable is, rather than leave it as the directory where
you launch mpirun from (if you launch spa_g++ on its own and not under
mpirun).  If that happens, SPARTA will look for additional input files
and write its output files to the executable directory, rather than
your working directory, which is probably not what you want.
</P>
<P>If SPARTA encounters errors in the input script or while running a
simulation it will print an ERROR message and stop or a WARNING
message and continue.  See <A HREF = "Section_errors.html">Section 12</A> for a
discussion of the various kinds of errors SPARTA can or can't detect,
a list of all ERROR and WARNING messages, and what to do about them.
</P>
<P>SPARTA can run a problem on any number of processors, including a
single processor.  The random numbers used by each processor will be
different so you should only expect statistical consistency if the
same problem is run on different numbers of processors.
</P>
<P>SPARTA can run as large a problem as will fit in the physical memory
of one or more processors.  If you run out of memory, you must run on
more processors or setup a smaller problem.
</P>
<HR>

<H4><A NAME = "start_6"></A>2.6 Command-line options 
</H4>
<P>At run time, SPARTA recognizes several optional command-line switches
which may be used in any order.  Either the full word or a one-or-two
letter abbreviation can be used:
</P>
<UL><LI>-e or -echo
<LI>-i or -in
<LI>-h or -help
<LI>-k or -kokkos
<LI>-l or -log
<LI>-p or -partition
<LI>-pk or -package
<LI>-pl or -plog
<LI>-ps or -pscreen
<LI>-sc or -screen
<LI>-sf or -suffix
<LI>-v or -var 
</UL>
<P>For example, spa_g++ might be launched as follows:
</P>
<PRE>mpirun -np 16 spa_g++ -v f tmp.out -l my.log -sc none < in.sphere
mpirun -np 16 spa_g++ -var f tmp.out -log my.log -screen none < in.sphere 
</PRE>
<P>Here are the details on the options:
</P>
<PRE>-echo style 
</PRE>
<P>Set the style of command echoing.  The style can be <I>none</I> or <I>screen</I>
or <I>log</I> or <I>both</I>.  Depending on the style, each command read from
the input script will be echoed to the screen and/or logfile.  This
can be useful to figure out which line of your script is causing an
input error.  The default value is <I>log</I>.  The echo style can also be
set by using the <A HREF = "echo.html">echo</A> command in the input script itself.
</P>
<PRE>-in file 
</PRE>
<P>Specify a file to use as an input script.  This is an optional switch
when running SPARTA in one-partition mode.  If it is not specified,
SPARTA reads its input script from stdin - e.g. spa_g++ < in.run.
This is a required switch when running SPARTA in multi-partition mode,
since multiple processors cannot all read from stdin.
</P>
<PRE>-help 
</PRE>
<P>Print a list of options compiled into this executable for each SPARTA
style (fix, compute, collide, etc).  SPARTA will print the info and
immediately exit if this switch is used.
</P>
<PRE>-kokkos on/off keyword/value ... 
</PRE>
<P>Explicitly enable or disable KOKKOS support, as provided by the KOKKOS
package.  Even if SPARTA is built with this package, as described
above in <A HREF = "#start_3">Section 2.3</A>, this switch must be set to enable
running with the KOKKOS-enabled styles the package provides.  If the
switch is not set (the default), SPARTA will operate as if the KOKKOS
package were not installed; i.e. you can run standard SPARTA 
for testing or benchmarking purposes.
</P>
<P>Additional optional keyword/value pairs can be specified which
determine how Kokkos will use the underlying hardware on your
platform.  These settings apply to each MPI task you launch via the
"mpirun" or "mpiexec" command.  You may choose to run one or more MPI
tasks per physical node.  Note that if you are running on a desktop
machine, you typically have one physical node.  On a cluster or
supercomputer there may be dozens or 1000s of physical nodes.
</P>
<P>Either the full word or an abbreviation can be used for the keywords.
Note that the keywords do not use a leading minus sign.  I.e. the
keyword is "t", not "-t".  Also note that each of the keywords has a
default setting.  Example of when to use these options and what
settings to use on different platforms is given in <A HREF = "Section_accelerate.html#acc_3">Section
5.3</A>.
</P>
<UL><LI>d or device
<LI>g or gpus
<LI>t or threads
<LI>n or numa 
</UL>
<PRE>device Nd 
</PRE>
<P>This option is only relevant if you built SPARTA with KOKKOS_DEVICES=Cuda, you
have more than one GPU per node, and if you are running with only one
MPI task per node.  The Nd setting is the ID of the GPU on the node to
run on.  By default Nd = 0.  If you have multiple GPUs per node, they
have consecutive IDs numbered as 0,1,2,etc.  This setting allows you
to launch multiple independent jobs on the node, each with a single
MPI task per node, and assign each job to run on a different GPU.
</P>
<PRE>gpus Ng Ns 
</PRE>
<P>This option is only relevant if you built SPARTA with KOKKOS_DEVICES=Cuda, you
have more than one GPU per node, and you are running with multiple MPI
tasks per node.  The Ng setting is how many GPUs
you will use per node.  The Ns setting is optional.  If set, it is the ID of a
GPU to skip when assigning MPI tasks to GPUs.  This may be useful if
your desktop system reserves one GPU to drive the screen and the rest
are intended for computational work like running SPARTA.  By default
Ng = 1 and Ns is not set.
</P>
<P>Depending on which flavor of MPI you are running, SPARTA will look for
one of these 4 environment variables
</P>
<PRE>SLURM_LOCALID (various MPI variants compiled with SLURM support)
MPT_LRANK (HPE MPI)
MV2_COMM_WORLD_LOCAL_RANK (Mvapich)
OMPI_COMM_WORLD_LOCAL_RANK (OpenMPI) 
</PRE>
<P>which are initialized by the "srun", "mpirun" or "mpiexec" commands.
The environment variable setting for each MPI rank is used to assign a
unique GPU ID to the MPI task.
</P>
<PRE>threads Nt 
</PRE>
<P>This option assigns Nt number of threads to each MPI task for
performing work when Kokkos is executing in OpenMP or pthreads mode.
The default is Nt = 1, which essentially runs in MPI-only mode.  If
there are Np MPI tasks per physical node, you generally want Np*Nt =
the number of physical cores per node, to use your available hardware
optimally. If SPARTA is compiled with KOKKOS_DEVICES=Cuda,
this setting has no effect.
</P>
<PRE>-log file 
</PRE>
<P>Specify a log file for SPARTA to write status information to.  In
one-partition mode, if the switch is not used, SPARTA writes to the
file log.sparta.  If this switch is used, SPARTA writes to the
specified file.  In multi-partition mode, if the switch is not used, a
log.sparta file is created with hi-level status information.  Each
partition also writes to a log.sparta.N file where N is the partition
ID.  If the switch is specified in multi-partition mode, the hi-level
logfile is named "file" and each partition also logs information to a
file.N.  For both one-partition and multi-partition mode, if the
specified file is "none", then no log files are created.  Using a
<A HREF = "log.html">log</A> command in the input script will override this setting.
Option -plog will override the name of the partition log files file.N.
</P>
<PRE>-partition 8x2 4 5 ... 
</PRE>
<P>Invoke SPARTA in multi-partition mode.  When SPARTA is run on P
processors and this switch is not used, SPARTA runs in one partition,
i.e. all P processors run a single simulation.  If this switch is
used, the P processors are split into separate partitions and each
partition runs its own simulation.  The arguments to the switch
specify the number of processors in each partition.  Arguments of the
form MxN mean M partitions, each with N processors.  Arguments of the
form N mean a single partition with N processors.  The sum of
processors in all partitions must equal P.  Thus the command
"-partition 8x2 4 5" has 10 partitions and runs on a total of 25
processors.  Note that with MPI installed on a machine (e.g. your
desktop), you can run on more (virtual) processors than you have
physical processors.
</P>
<P>To run multiple independent simulatoins from one input script, using
multiple partitions, see <A HREF = "Section_howto.html#howto_3">Section 6.3</A> of
the manual.  World- and universe-style variables are useful in this
context.
</P>
<PRE>-package style args .... 
</PRE>
<P>Invoke the <A HREF = "package.html">package</A> command with style and args.  The
syntax is the same as if the command appeared at the top of the input
script.  For example "-package kokkos on gpus 2" or "-pk kokkos g 2" is the same as
<A HREF = "package.html">package kokkos g 2</A> in the input script.  The possible styles
and args are documented on the <A HREF = "package.html">package</A> doc page.  This
switch can be used multiple times.
</P>
<P>Along with the "-suffix" command-line switch, this is a convenient
mechanism for invoking the KOKKOS accelerator package and its options without
having to edit an input script.
</P>
<PRE>-plog file 
</PRE>
<P>Specify the base name for the partition log files, so partition N
writes log information to file.N. If file is none, then no partition
log files are created.  This overrides the filename specified in the
-log command-line option.  This option is useful when working with
large numbers of partitions, allowing the partition log files to be
suppressed (-plog none) or placed in a sub-directory (-plog
replica_files/log.sparta) If this option is not used the log file for
partition N is log.sparta.N or whatever is specified by the -log
command-line option.
</P>
<PRE>-pscreen file 
</PRE>
<P>Specify the base name for the partition screen file, so partition N
writes screen information to file.N. If file is none, then no
partition screen files are created.  This overrides the filename
specified in the -screen command-line option.  This option is useful
when working with large numbers of partitions, allowing the partition
screen files to be suppressed (-pscreen none) or placed in a
sub-directory (-pscreen replica_files/screen) If this option is not
used the screen file for partition N is screen.N or whatever is
specified by the -screen command-line option.
</P>
<PRE>-screen file 
</PRE>
<P>Specify a file for SPARTA to write its screen information to.  In
one-partition mode, if the switch is not used, SPARTA writes to the
screen.  If this switch is used, SPARTA writes to the specified file
instead and you will see no screen output.  In multi-partition mode,
if the switch is not used, hi-level status information is written to
the screen.  Each partition also writes to a screen.N file where N is
the partition ID.  If the switch is specified in multi-partition mode,
the hi-level screen dump is named "file" and each partition also
writes screen information to a file.N.  For both one-partition and
multi-partition mode, if the specified file is "none", then no screen
output is performed. Option -pscreen will override the name of the 
partition screen files file.N.
</P>
<PRE>-suffix style args 
</PRE>
<P>Use variants of various styles if they exist.  The specified style can
be <I>kk</I>.  This refers to optional KOKKOS package that SPARTA can be built with, as described
above in <A HREF = "#start_3">Section 2.3</A>.
</P>
<P>Along with the "-package" command-line switch, this is a convenient
mechanism for invoking the KOKKOS accelerator package and its options without
having to edit an input script.
</P>
<P>As an example, the KOKKOS package provides a <A HREF = "compute_temp.html">compute_style
temp</A> variant, with style name temp/kk. A variant style
can be specified explicitly in your input script, e.g. compute
temp/kk. If the suffix command is used with the appropriate style,
you do not need to modify your input script.  The specified suffix
(kk) is automatically appended whenever your
input script command creates a new <A HREF = "fix.html">fix</A>,
<A HREF = "compute.html">compute</A>, etc.
If the variant version does not exist, the standard version is
created.
</P>
<P>For the KOKKOS package, using this command-line switch also invokes
the default KOKKOS settings, as if the command "package kokkos" were
used at the top of your input script.  These settings can be changed
by using the "-package kokkos" command-line switch or the <A HREF = "package.html">package
kokkos</A> command in your script.
</P>
<P>The <A HREF = "suffix.html">suffix</A> command can also be used within an input
script to set a suffix, or to turn off or back on any suffix setting
made via the command line.
</P>
<PRE>-var name value1 value2 ... 
</PRE>
<P>Specify a variable that will be defined for substitution purposes when
the input script is read.  "Name" is the variable name which can be a
single character (referenced as $x in the input script) or a full
string (referenced as ${abc}).  An <A HREF = "variable.html">index-style
variable</A> will be created and populated with the
subsequent values, e.g. a set of filenames.  Using this command-line
option is equivalent to putting the line "variable name index value1
value2 ..."  at the beginning of the input script.  Defining an index
variable as a command-line argument overrides any setting for the same
index variable in the input script, since index variables cannot be
re-defined.  See the <A HREF = "variable.html">variable</A> command for more info on
defining index and other kinds of variables and <A HREF = "Section_commands.html#cmd_2">Section
3.2</A> for more info on using variables in
input scripts.
</P>
<P>IMPORTANT NOTE: Currently, the command-line parser looks for arguments
that start with "-" to indicate new switches. Thus you cannot specify
multiple variable values if any of they start with a "-", e.g. a
negative numeric value. It is OK if the first value1 starts with a
"-", since it is automatically skipped.
</P>
<HR>

<H4><A NAME = "start_7"></A>2.7 SPARTA screen output 
</H4>
<P>As SPARTA reads an input script, it prints information to both the
screen and a log file about significant actions it takes to setup a
simulation.  When the simulation is ready to begin, SPARTA performs
various initializations and prints the amount of memory (in MBytes per
processor) that the simulation requires.  It also prints details of
the initial state of the system.  During the run itself, statistical
information is printed periodically, every few timesteps.  When the
run concludes, SPARTA prints the final state and a total run time for
the simulation.  It then appends statistics about the CPU time and
size of information stored for the simulation.  An example set of
statistics is shown here:
</P>
<P>Loop time of 0.639973 on 4 procs for 1000 steps with 45792 particles
</P>
<PRE>MPI task timing breakdown:
Section |  min time  |  avg time  |  max time  |%varavg| %total
---------------------------------------------------------------
Move    | 0.10948    | 0.26191    | 0.42049    |  27.6 | 40.92
Coll    | 0.013711   | 0.041659   | 0.070985   |  13.5 |  6.51
Sort    | 0.01733    | 0.040286   | 0.063573   |  10.6 |  6.29
Comm    | 0.02276    | 0.023555   | 0.02493    |   0.6 |  3.68
Modify  | 0.00018167 | 0.024758   | 0.051345   |  15.6 |  3.87
Output  | 0.0002172  | 0.0007354  | 0.0012152  |   0.0 |  0.11
Other   |            | 0.2471     |            |       | 38.61 
</PRE>
<PRE>Particle moves    = 38096354 (38.1M)
Cells touched     = 43236871 (43.2M)
Particle comms    = 146623 (0.147M)
Boundary collides = 182782 (0.183M)
Boundary exits    = 181792 (0.182M)
SurfColl checks   = 7670863 (7.67M)
SurfColl occurs   = 177740 (0.178M)
Surf reactions    = 124169 (0.124M)
Collide attempts  = 1232 (1K)
Collide occurs    = 553 (0.553K)
Gas reactions     = 23 (0.023K)
Particles stuck   = 0 
</PRE>
<PRE>Particle-moves/CPUsec/proc: 1.4882e+07
Particle-moves/step: 38096.4
Cell-touches/particle/step: 1.13493
Particle comm iterations/step: 1.999
Particle fraction communicated: 0.00384874
Particle fraction colliding with boundary: 0.00479789
Particle fraction exiting boundary: 0.0047719
Surface-checks/particle/step: 0.201354
Surface-collisions/particle/step: 0.00466554
Surface-reactions/particle/step: 0.00325934
Collision-attempts/particle/step: 1.232
Collisions/particle/step: 0.553
Gas-reactions/particle/step: 0.023 
</PRE>
<P>Gas reaction tallies:
  style tce #-of-reactions 45
  reaction O2 + N --> O + O + N: 10
  reaction O2 + O --> O + O + O: 5
  reaction N2 + O --> N + N + O: 8
</P>
<P>Surface reaction tallies:
  id 1 style global #-of-reactions 2
    reaction all: 124025
    reaction delete: 53525
    reaction create: 70500
</P>
<PRE>Particles: 11448 ave 17655 max 5306 min
Histogram: 2 0 0 0 0 0 0 0 0 2
Cells:     100 ave 100 max 100 min
Histogram: 4 0 0 0 0 0 0 0 0 0
GhostCell: 21 ave 21 max 21 min
Histogram: 4 0 0 0 0 0 0 0 0 0
EmptyCell: 21 ave 21 max 21 min
Histogram: 4 0 0 0 0 0 0 0 0 0
Surfs:     50 ave 50 max 50 min
Histogram: 4 0 0 0 0 0 0 0 0 0
GhostSurf: 0 ave 0 max 0 min
Histogram: 4 0 0 0 0 0 0 0 0 0 
</PRE>
<P>The first line gives the total CPU run time for the simulation, in
seconds.
</P>
<P>The next section gives a breakdown of the CPU timing (in seconds) in
7 categories.  The first four are timings for particles moves, which
includes interaction with surface elements, then particle collisions,
then sorting of particles (required to perform collisions), and
communication of particles between processors.  The Modify section is
time for operations invoked by fixes and computes.  The Output section
is for dump command and statistical output.  The Other category is
typically for load-imbalance, as some MPI tasks wait for others MPI
tasks to complete.  In each category the min,ave,max time across
processors is shown, as well as a variation, and the percentage of
total time.
</P>
<P>The next section gives some statistics about the run.  These are total
counts of particle moves, grid cells touched by particles, the number
of particles communicated between processors, collisions of particles
with the global boundary and with surface elements (none in this
problem), as well as collision and reaction statistics.
</P>
<P>The next section gives additional statistics, normalized by timestep
or processor count.
</P>
<P>The next 2 sections are optional.  The "Gas reaction tallies" section
is only output if the <A HREF = "react.html">react</A> command is used.  For each
reaction with a non-zero tally, the number of those reactions that
occurred during the run is printed.  The "Surface reaction tallies"
section is only output if the <A HREF = "surf_react.html">surf_react</A> command was
used one or more times, to assign reaction models to individual
surface elements or the box boundaries.  For each of the commands, and
each of its reactions with a non-zero tally, the number of those
reactions that occurred during the run is printed.  Note that this is
effectively a summation over all the surface elements and/or box
boundaries the <A HREF = "surf_react.html">surf_react</A> command was used to assign
a reaction model to.
</P>
<P>The last section is a histogramming across processors of various
per-processor statistics: particle count, owned grid cells, processor,
ghost grid cells which are copies of cells owned by other processors,
and empty cells which are ghost cells without surface information
(only used to pass particles to neighboring processors).
</P>
<P>The ave value is the average across all processors.  The max and min
values are for any processor.  The 10-bin histogram shows the
distribution of the value across processors.  The total number of
histogram counts is equal to the number of processors.
</P>
</HTML>
